{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [10、PyTorch autograd使用教程](https://www.bilibili.com/video/BV1vL411u7bL?spm_id_from=333.788.player.switch&vd_source=cdd897fffb54b70b076681c3c4e4d45d)\n",
    "\n",
    "---\n",
    "\n",
    "两个教程：\n",
    "\n",
    "- [Automatic differentiation in machine learning: a survey](https://arxiv.org/abs/1502.05767)\n",
    "- cs321 中的PPT：\n",
    "\n",
    "![image-20250619024650753](http://assets.hypervoid.top/img/2025/06/19/image-20250619024650753-51f3.png)\n",
    "\n",
    "![image-20250619025001840](http://assets.hypervoid.top/img/2025/06/19/image-20250619025001840-f0f3.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torchvision.models import resnet18, ResNet18_Weights\n",
    "model = resnet18(weights=ResNet18_Weights.DEFAULT)\n",
    "data = torch.rand(1, 3, 64, 64)\n",
    "labels = torch.rand(1, 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<SumBackward0 at 0x7f229247bc10>, <AddBackward0 at 0x7f2292478460>)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from torch import binary_cross_entropy_with_logits\n",
    "torch.manual_seed(1111)\n",
    "x = torch.ones(5)\n",
    "y = torch.zeros(3)\n",
    "w = torch.randn(5,3, requires_grad=True)\n",
    "b = torch.randn(3, requires_grad=True)\n",
    "z = torch.matmul(x, w) + b\n",
    "loss = binary_cross_entropy_with_logits(z,y)\n",
    "loss = binary_cross_entropy_with_logits(z,y).sum()\n",
    "loss.backward()# 如果不带参数，loss必须是一个标量\n",
    "# loss.backward(torch.ones(3)) # 带参数传入一个张量\n",
    "# 不能两次调用 backward，因为计算图已经消失了\n",
    "loss.grad_fn, z.grad_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0.0357, 0.5151, 0.5091],\n",
       "         [0.0357, 0.5151, 0.5091],\n",
       "         [0.0357, 0.5151, 0.5091],\n",
       "         [0.0357, 0.5151, 0.5091],\n",
       "         [0.0357, 0.5151, 0.5091]]),\n",
       " tensor([0.0357, 0.5151, 0.5091]))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w.grad, b.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输入x计算出z，和y算出loss,loss反向传播更新w和b\n",
    "![image-20250619025339945](http://assets.hypervoid.top/img/2025/06/19/image-20250619025339945-fe8a.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## no_grad\n",
    "\n",
    "计算梯度消耗很大，在不训练，推理的时候，不计算梯度比较好。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = resnet18(weights=ResNet18_Weights.DEFAULT)\n",
    "# with torch.no_grad():\n",
    "    # model(input)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 梯度清零\n",
    "torch中梯度默认会积累，因此在optimizer中经常运行 `zero_grad_()`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
